The code of modern software, including malware, constitutes a complex set of instructions for jumps, calls, loops and other operations. It should be noted that the executable files are becoming increasingly complex due to the increasing popularity of high-level programming languages and also the greater complexity of computer hardware and operating systems. This applies both to trusted programs and to malicious ones. Malicious programs can perform a number of characteristic actions, such as stealing passwords and other confidential user data, adding a computer to a bot network to carry out DDoS attacks or send out spam mail, blocking the correct operations of the system for purposes of extortion, and other malicious actions that are unwanted by the user and harmful to the user's computer system.
One of the ways of investigating potentially harmful programs is by antivirus applications that may use a program emulator to analyzing the behavior of the programs. There are various methods of emulation. One of them is a program imitation of the processor, the memory, and other devices by creating virtual copies of the registers of the processor, memory, and instruction set of the processor. Thus, the program instructions are executed not on the real processor, but on its virtual copy, and the calls of the system API functions are emulated and the emulated result of the working of the function is sent in response.
It should be noted that the process of initializing an emulator can be rather resource-intensive. The initialization of the emulator should involve not only creating a virtual copy of the necessary hardware (processor, working memory), but also virtual copies of a number of key components of the operating system (OS) in which the emulation of the execution of the application will occur. Among the key components of the OS one can mention, for example, the kernel of the operating system, which is responsible for its necessary mechanisms of operation, such as the processing of interrupts and exceptions, core drivers, memory manager, and so forth. For a proper “mimicking” (emulating) of the working OS, one needs to repeat the process of its loading, albeit in a rather simplified form.
FIG. 1 shows the process of loading the Windows operating system. In step 101, Windows Boot Manager may be loaded, being responsible for finding the installed OS Windows and enabling the selecting of the loading of any OS found. Next, in step 102, the loading of the basic drivers may occur, which are responsible, for example, for enabling a working with the hard disk partition on which the selected OS is installed (i.e., the file system driver is loaded). Next, the necessary part of OS kernel may be read from the disk and loaded into memory in step 103, for example, Ntoskrnl.exe and hal.dll, and the registry, memory manager, object manager, etc., are initialized. In step 104, the sessions manager (smss.exe) is loaded, which is responsible for loading the system variables, the Win32 subsystem, and the further loading of winlogon.exe in step 105. After the user successfully performs the authentication in step 106, the applications and services with autorun registry key are loaded, and then the OS will be fully ready to interact with the user, awaiting the starting of applications and the input of data.
For the emulation process, it is not necessary to fully emulate the loading of the operating system. For example, steps 101 and 102 can be omitted, leaving only steps 103 and 104 in simplified form. That is, it will be sufficient to emulate the most important functionality for the correct functioning of the OS, which is required for application emulation. For Win32 applications, it will be necessary to emulate the starting of smss.exe with subsequent starting of csrss.exe, which initializes the Windows subsystem and enables creating of processes and threads. Since the emulation of a potentially harmful application requires the creating of a more detailed operating environment (such as the emulation of other processes or services), it is also necessary to emulate the starting of winlogon.exe with the subsequent “starting” of processes like explorer.exe and services.exe, it being possible to emulate the starting of svchost processes from the latter. By the term “starting” in this case is meant the recreating in the emulator of the same processes that occur when the processes are created within the real OS, albeit in greatly simplified form. Such an approach lets one recreate the real OS to a sufficient degree for starting practically any application designed to work in the given OS. For potentially harmful applications, such a level of detail in recreating the environment is also necessary, both in order to get around possible anti-emulation tricks that might include the actual checking for the presence of started services, system variables values, and other elements which are present in the real OS, and in order for the malicious functional to be implemented, which may be aimed at certain applications. As an example, one can mention the process explorer.exe, which is often the target of attack, the web browser processes, for which a corresponding exploit can be utilized, and so on.
Thus, the problem of creating a corresponding environment for emulation of a potentially harmful application requires the most detailed possible recreating of the real OS and the applications started in it. The process of initialization of such an environment can take up much time and resources (starting of the emulation process, loading from hard disk into memory all necessary data for the initialization of such virtual structures as the file system, the registry, and so on), which shortens the time for the actual emulation of the application code.
At present, approaches exist for preserving the state of an emulated environment. For example, the emulator QEMU supports the creation of images, including those which preclude the need to perform the process of loading of the OS. However, the image is preserved on disk only at the request of the user, and it does not support any structure for preserving a series of recorded images. This approach concurs entirely with the operating mode of a virtual machine manager.
However, the known approaches do not involve automatic creation of images of an emulated system, or the state of execution of an emulated application that would make it possible to track down every possible branching of the execution of the program code. From the standpoint of the analysis of malicious programs, such an approach would be able to get around anti-emulation tricks.
The creators of malware use various tricks to defeat emulation of the program code, which can be based on limitations relating to the emulation process and the implementing of an emulator in antivirus solutions. The emulator creates a virtual copy of the processor, the computer components and the operating system (OS) only to a limited extent with curtailed capabilities, since the full recreation of all capabilities of such a processor or system API functions is not possible for a number of reasons: the immense labor for such a development, the presence of undocumented functions, the large drop in performance when such an emulator is working. Thus, the creators of malware can resort to the following approaches for detecting the fact of an execution in an emulated environment:                The calling of an undocumented or seldom used API function.        The execution of a number of instructions of the central processor with subsequent checking, for example, of a number of flags that are set. If the emulation of the processor commands is insufficiently precise, a number of flags might have values different from those which would have been set when executed by real processor.        Checking of the correct execution of an API function. The checking can be very complicated and involve an analysis of the error codes returned during an incorrect call or a checking of the processor registers values.        Looking for certain bytes in memory. For example, a byte by byte search for the header MZ in memory after the loading of kernel32.dll when starting the process. In the OS Vista 64, a 64 kb alignment is used for the kernel32.dll, and the region between the header and the first section will not be mapped in the address space of the process. If an access to this is attempted, an exception will occur. If the exception was not registered, the standard OS exceptions handler will be called up, and this will terminate the process.        